Add parallel setup driver for GCP resources - #67
Merged
Conversation
Runs setup_composer.sh, setup_feature_store.sh and setup_artifact_registry.sh concurrently and prints a status table. Wall clock becomes Composer's ~25 min rather than the sum of all three. Mirrors destroy_resources.sh in flags and structure: --skip-composer / --skip-feature-store / --skip-artifact-registry --serial, --dry-run, --timeout, --heartbeat, --log-dir Details: - Each script writes to its own log file; three interleaved gcloud streams on one terminal are unreadable. Failed logs are tailed in the summary. - Not `set -e` — a failing child must be recorded and reported alongside the others, not abort the driver and hide the rest. Exit 1 if any target failed. - Pre-flight checks gcloud auth and that feature_store is importable, so a missing venv fails in one second rather than 25 minutes into a Composer build. - Skipped targets appear in the summary; omitting them would read as "all succeeded" when only one ran. - bash 3.2 compatible (macOS /bin/bash): indexed arrays, no `wait -n`. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds
scripts/setup_resources.sh, which runs the three setup scripts concurrently and reports a status table. Wall clock becomes Composer's ~25 min instead of the sum of all three.Mirrors
destroy_resources.shin flags and structure, so the create/destroy pair stays symmetric.Usage
Output
Exit code is 0 only if every target that actually ran succeeded.
Design notes
gcloudoutput on one terminal is unreadable. Failed logs are tailed automatically in the summary.set -e. A failing child script has to be recorded and reported next to the others, not abort the driver and hide the rest.feature_storeis importable. Without them, a missing venv fails feature-store one second in while Composer keeps building for 25 minutes./bin/bash): indexed arrays only, nowait -n, no associative arrays.--timeoutkills the process group rather than the parent, sincegcloudspawns children that would otherwise outlive it. Off by default — interrupting a half-created Composer environment is worse than waiting.Testing
Exercised against stub scripts covering: all-success, partial failure with exit-code propagation, skip reporting, serial mode, timeout watchdog, dry-run, unknown-argument rejection, all-skipped, and pre-flight pass. Not yet run against live GCP.
🤖 Generated with Claude Code